CS 229 Final Project: Single Image Depth Estimation From Predicted Semantic Labels

Recovering the 3D structure of a scene from a single image is a fundamental problem in computer vision that has applications in robotics, surveillance, and general scene understanding. However, estimating structure from raw image features is notoriously difficult since local appearance is insufficient to resolve depth ambiguities (e.g., sky and water regions in an image can have similar appearance but dramatically different geometric placement within a scene). Intuitively, semantic understanding of a scene plays an important role in our own perception of scale and 3D structure. Our goal is to estimate the depth of each pixel in an image. We employ a two phase approach: In the first phase, we use a learned multi-class image labeling MRF to estimate the semantic class for each pixel in the image. We currently label pixels as one of: sky, tree, road, grass, water, building, mountain, and foreground object. In the second phase, we use the predicted semantic class labels to inform our depth reconstruction model. Here, we first learn a separate depth estimator for each semantic class. We incorporate these predictions in a Markov random field (MRF) that includes semantic-aware reconstruction priors such as smoothness and orientation. Motivated by the work of Saxena et. al., [6], we explore both pixelbased and superpixel-based variants of our model.